RS-Algo: an Algorithm for Improved Memory Utilization in Continuous Query System under Asynchronous Data Streams

Continuous query systems are an intuitive way for users to access data streaming data in large scale scientific applications containing hundreds if not thousands of streams. This paper focuses on optimizations for improved memory utilization under the large scale, asynchronous streams found in on-demand meteorological forecasting and other data-driven applications. Specifically, we experimentally evaluate the RS-Algo algorithm for dynamically adjusting sliding window sizes used in database joins to reflect current stream rates. In this TR we provide detailed results of measurements of the RS-Algo algorithm which dynamically adjusts the sliding window sizes used in database joins to reflect current stream rates. We tested under various queries and stream rate conditions using a synthetic workloads and using the dQUOB continuous query system. We examine service time and memory utilization. The second result chronicles a smaller experiment comparing two algorithms used during runtime to schedule events. We evaluated the performance of the existing algorithm, First Come First Serve (FCFS), against a modified version of Earliest Job First (EJF) and examined the impact on memory utilization.